This invention relates generally to the fields of aerial surveillance and mapping from aerial photography. In particular, the invention relates to a method of determining the geo-location (e.g., longitude and latitude coordinates) of any arbitrary point in a photograph of the Earth's surface taken by a digital camera in an airborne vehicle or satellite. The method is particularly useful for geo-location from oblique imagery.
The known prior art includes several references related to the problem of locating coordinates of a selected point on the ground within imagery obtained from an airborne camera. The prior art of interest includes U.S. Pat. No. 6,735,348 to Dial, Jr. et al.; U.S. Pat. No. 5,633,946 to Lachinski et al., U.S. Pat. No. 7,233,691 to Setterholm; and U.S. Pat. No. 5,596,494 to Kuo.
Prior art methodologies typically accomplish the task by calculating the location of the airborne platform using the onboard inertial navigation system (INS), followed by one of the two means: 1) Parametric Model: constructing a parametric representation of the camera and solving the coefficients in the mathematical model by using the airborne platform locations and known ground control points (GCP) in multiple captured images. Once the mathematical model is obtained and fine tuned, the ground coordinates of a point in any photograph taken subsequently can be derived by piping the coordinates of the input point and the platform location through the mathematical model, and 2) Physical Model: referencing the airborne platform location to the image detector focal plane array within the camera, and then linearly projecting the captured airborne image represented by the pixels of the array onto the ground space. The ground location of an image pixel can be obtained by this light ray tracing method, which relies on the physical properties of the camera, its location and line of sight when the image is collected. Typically, the ground geo-locations of the four corners of images are calculated during image acquisition and annotated for post-processing. With the four corner geo-locations, a linear interpolation method is then used to find the ground locations of selected interior points in the image, represented by the corresponding pixels on the detector array. The linear interpolation method scales the image space detector pixels with a pair of linear factors in the longitude and latitude directions to find the coordinates of the pixels proportionally in the ground space. While this method gives satisfactory results for imagery captured vertically or perpendicular (at nadir) to the platform, it produces significant error when imagery is captured at angles forward oblique or side oblique relative to the platform. This error is due to the progressive scaling characteristics intrinsic to oblique images, and the application of linear interpolation techniques introduces approximation errors along the vanishing lines.
The error produced by using linear interpolation to geo-locate objects in oblique imagery stems from the fact that the field of view (FOV) of an obliquely captured image can be described as having a trapezoidal shape relative to the ground, when defined by the far and near corner points. However, the FOV of imagery captured at nadir appears square shaped since the near and far corner points are equidistant from the center of the image. As the angle of capture relative to the platform nadir increases (i.e., the angle is progressively oblique), the error contained in coordinates of a linearly projected image point on the ground also increases. Thus, it can be demonstrated that using the linear interpolation method of the prior art to map obliquely captured airborne image space onto ground space for the purpose of locating an object point on the ground produces errors in geo-location, with the magnitude of the errors increasing as the angle becomes more oblique.
Another prior art method of performing geo-location from imagery is a warping technique in which images are warped to a world coordinate space such as WGS 84. In this method, a mapping is performed of the image either linearly or non-linearly with warping to a display space representing the world coordinate space and then performing linear interpolation in the display space. When non-linear algorithms are used, the resulting coordinates are more precise. However, this technique requires image warping, which distorts objects and scenes in the image and is computationally time consuming.
Another method is a raytracing technique that requires the use of a camera model that defines and simulates the characteristics of the camera. In this method, the camera optical and geometrical model is used to derive the projected location from the image pixel in the focal plane to the point on the ground. In addition to the camera model, this technique also requires the location (longitude, latitude, and altitude) of the camera and use of a certain Earth model (Flat, Spherical, Ellipsoid, etc.). The need for the above information greatly limits the usefulness of this method for general application to oblique imagery because the camera model for each image being exploited needs to be identified.
To precisely acquire the geo-locations of four corners and/or any selected point from imagery, another method is described in the patent application of Reneker, et al., U.S. Ser. No. 11/222,562 titled “Precision Optical Systems with Performance Characterization and Uses Thereof” which is assigned to the assignee of present applicant, Recon/Optical Inc. Reneker et al. describes a method for locating a point on the ground within the airborne captured imagery by first characterizing, in real time, the overall optical system which has captured the imagery. The characterization data is then used to correct the optical projection of the light rays of any ground point in the imagery onto an image detector focal plane array which has had its pixel elements accurately referenced to inertial space coordinates by an onboard inertial measurement unit and a beam collimator which projects calibration points onto the detector array. This method accurately accomplishes the location task totally within the airborne platform.
Other prior art of interest include the following text books: Samet, H., “Quad-Tree Representations of Image Data: The Design and Analysis of Spatial Data Structures”, 1990, Addison-Wesley, Reading, Mass., and Samet, H., “Applications of Spatial Data Structures: Computer Graphics, Image Processing, and GIS”, 1990, Addison-Wesley, Reading, Mass. These books describe quad-tree and oct-tree structures for representing 2-dimensional and 3-dimensional objects in raster (pixel or voxel) formats. The applications shown in the books use such structures in computer data representations for performing data compression, window clipping, linear image transformations, raytracing for image rendering, and finding distances among tree nodes. The associated technique of quad-tree partitioning of images, where images are divided into quadrants for subsequent image processing, is a common practice in computer graphics. However, the application of quad-tree partitioning in parallel for comparing images in image space (discrete, integer) to oblique images in ground space (continuous, real) in order to find and match locations in each space, as described in the present invention, is believed to be entirely novel and nonobvious.
Current reconnaissance cameras used in military applications supply imagery to a ground station in a data format that is known as NITF (National Imagery Transmission Format). The National Imagery Transmission Format Standard (NITFS) is a suite of standards for formatting digital imagery and imagery-related products and exchanging them among the Department of Defense, other intelligence community members, and other United States Government departments and agencies. Resulting from a collaborative US Government and industry effort, it is the common standard used to exchange and store files composed of images, symbols, text, and associated data. Persons skilled in this art are familiar with the NITF standards. In practice, a reconnaissance camera in compliance with NITF includes an image processing unit that computes the longitude and latitude coordinates of the four corner points of the captured ground space image using known information such as camera location and orientation, depression angle, altitude, and field of view. The image processing unit then inserts the calculated ground coordinates of the image corner points into the NITF data header for a ground station to read and use in operation, e.g., by a geo-location algorithm to locate objects within the imagery as described in the prior art and in this disclosure.
The present invention provides a more accurate method for geo-locating points of interest in oblique aerial reconnaissance imagery than those methods of the prior art which are based on linear interpolation. Furthermore, the method described in the present invention is particularly useful in that it does not require characterization of the camera optical system or use of a camera model and is applicable to any digital image captured at an oblique angle where the geo-location of the corner points in the image is known, e.g., in images annotated in accordance with the NITF standard.